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ABSTRACT 



The relationship between selected contextual and 
socioeconomic variables and school -level results from a statewide 
performance-based student assessment system was studied using the Kentucky 
Instructional Results Information System (KIRIS) . KIRIS uses portfolios of 
student work, scores from on-demand assessments, and student performance on 
performance events in the areas of mathematics, science, social studies, arts 
and humanities, practical living, and vocational skills. Multiple regression 
analyses were conducted using data gathered over a 3-year period (1993-1995) 
from 49 schools in one school district. Results show that a single 
socioeconomic variable, the percent of students qualifying for free or 
reduced lunch benefits, explains a large portion of the variance in scores at 
all school levels. Contextual or socioeconomic indicators were not, however, 
predictive of improvements in scores from year to year. Although 
socioeconomic indicators explain much of the variation in elementary, middle, 
and high schools' levels of achievement, they do not explain the variation in 
score improvements made by schools in this district. Possible explanations 
for these results and school level differences are discussed, along with 
implications for performance-based student assessment systems. (Contains 5 
tables and 21 reference.) (Author/SLD) 
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Abstract 



The purpose of this study was to investigate the relationship between selected contextual 
and socioeconomic variables and school-level results from a statewide, performance-based student 
assessment system. Multiple regression analyses were conducted using data gathered over a three 
year period from 49 schools in one school district. Results showed that a single socioeconomic 
variable, the percent of students qualifying for free or reduced lunch benefits, explained a large 
portion of the variation in scores at all school levels. Contextual or socioeconomic indicators were 
not, however, predictive of improvements in scores from year to year. Possible explanations for 
these results and school level differences are discussed, along with implications for performance- 
based student assessment systems. 
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The Relationship Between Socioeconomic Characteristics 
and School-Level Performance Assessment Results 



Performance assessment is a dominant force in education today. Spurred by evidence of the 
serious limitations and numerous unintended, negative consequences of standardized, multiple- 
choice tests, performance assessments are advocated as far more “authentic” measures of student 
learning (Meisels, Dorfman, & Steele, 1995; Wiggins, 1993). When compared to traditional tests, 
performance assessments appear less stigmatizing, more instructionally relevant, more adaptable to 
individual student differences, more faithful to the richness and complexity of real-world problem- 
solving, and more reflective of the actual quality of student understanding. As Bond (1995, p . 21) 
points out, however, these qualities are more “articles of faith” than demonstrated characteristics. 
Little evidence exists currently that performance assessments are truly less stigmatizing or that they 
are a more accurate reflection of student learning. 

Another suggested advantage of performance assessments is that they are, at least 
potentially, less biased and more fair to traditionally disadvantaged students. This is because 
performance assessments, when properly used, can merge instruction and assessment, rather than 
simply test abilities (such as recall of decontextualized information) that are only remotely 
connected to the everyday experiences of most students (Simmons & Resnick, 1993). Although it 
would be premature to say that any definitive conclusion has been reached regarding this claim, the 
information gathered to date is not particularly promising. Jaeger (1992), for example, reports a 
rather consistent pattern of findings linking poverty rate and other family structure variables to 
student performance on problem solving tasks in international mathematics assessments. Similarly, 
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analyses of data from the National Assessment of Educational Progress show that after correcting 
for reliability, differences in scores on extended-response essays between African American 
students and White students actually exceed those on the multiple-choice reading assessment 
(Baker, O’Neil, & Linn, 1991; Linn, Baker, & Dunbar, 1991). 

This study was designed to provide additional insights into this complex interaction. 
Specifically, it’s purpose was to investigate the relationship between selected contextual and 
socioeconomic variables, and school-level results from a statewide performance-based assessment 
system. We sought to determine the degree to which school-level performance assessment results 
could be predicted from data on selected contextual factors and the socioeconomic characteristics 
of a school’s student population. A previous study by Guskey and Kifer (1990) showed that 
certain socioeconomic indices could be used to explain a large portion of the variation 
(approximately 42%) among school districts in the results from a basic skills testing program used 
formerly in the same state. The developers of the current performance-based assessment system 
hoped that the change in assessment content, format, and procedures might alter this trend. 

The Assessment 

A major element of the Kentucky Education Reform Act is a statewide student assessment 
program known as the Kentucky Instructional Results Information System, or KIRIS. This multi- 
faceted performance assessment system has three components (see Guskey, 1994a). The first is 
evaluations of portfolios of students’ work in the areas of writing, mathematics, science, and social 
studies. The second is based on students’ scores from “on-demand assessments” in the areas of 
reading, mathematics, science, social studies, arts and humanities, practical living and vocational 
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skills. These assessments include both multiple-choice and open-ended items similar to those used 
in the National Assessment of Educational Progress. The third component involves students’ 
performance on a series of “performance events” in the areas of mathematics, science, social 
studies, arts and humanities, practical living and vocational skills. 

Results from these three components of KIRIS are combined with a series of “non-cognitive 
indicators” (e.g., school attendance, retention, dropout rate, etc.) to calculate an “accountability 
index score” for every public school in the Commonwealth. In determining the “accountability 
index” in the earliest versions of KIRIS, performance assessment results contribute 5/6th while the 
“non-cognitive indicators” contribute l/6th. 

Scores on the “accountability index” can range from 0 to 100, and are based on the goal of 
having all students (100%) score at the “proficient” level on the assessments within a period of 20 
years (Kifer, 1994). In practice, however, not all students need to score at the “proficient” level 
for a school to reach an accountability index score of 100. Students who score at the highest 
“distinguished” level on the assessments balance the scores of those students who score at the 
lower levels of “novice” and “apprentice.” 

The first year the KIRIS assessments were administered in 1992, results were used to 
determine a “baseline” accountability index score for each school. From this baseline, all schools 
were expected to show improvement. The amount of improvement expected was determined by 
calculating a “threshold” score for each school. Thresholds were established by first computing the 
gain a school would have to make to reach an accountability index score of 100 within a time 
period of 20 years. This total amount is then divided into ten, equal increments. Adding one 
increment to a school’s “baseline” yields that school’s “threshold” for the next biennial assessment. 
So, for example, if a school’s “baseline” accountability index score in 1992 was 30, it’s “threshold” 
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for 1994 would be 37 ([100-30J/10 = 7; 30 + 7 — 37). Based on a school’s results, new 
thresholds are then recalculated for the next assessment cycle. 

To avoid the problems associated with year to year score fluctuations (Mandeville & 
Khosrow, 1988), scores from two years are combined (averaged) to obtain each school’s new 
accountability index score. This score is then compared to the school’s “threshold” to determine if 
expected progress has been made. In other words, the accountability associated with KIRIS is not 
based on achievement per se; rather, it is based on improvement. Each school is compared to itself 
and is expected to show consistent improvement over time. 

The reform legislation further specifies that results from the assessments will be used to 
grant financial rewards to schools that show significant improvements (i.e., make gains beyond their 
“threshold”) and to levy sanctions against those that fail to show progress. In other words, the 
assessment program is also “high stakes” (Trimble, 1994). 

Data Source 

The principal source of data for this study was the 1993, 1994, and 1995 KIRIS results 
from one of the largest school districts in the Commonwealth. This county school district includes 
a metropolitan area and its surrounding suburbs. Nearly 34,000 students are enrolled in the 
district’s 33 elementary schools (grades K-5), 1 1 middle schools (grades 6-8), and 5 high schools 
(grades 9-12). The performance data used in this study include the accountability index scores for 
each of the 49 schools in the district for each of the three years. Simple gain scores also were 
calculated by subtracting the previous years’ score from each school’s accountability index score. 

The reform legislation originally specified that assessment results from grades 4, 8, and 12 
would be used to calculate the accountability index scores for elementary, middle, and high schools. 




6 



respectively. But several changes have been made to this policy since the original legislation was 
passed. Specifically, the mathematics portfolio assessment has been moved from grade 4 to grade 
5, and the performance assessments and on-demand assessments for high school students have been 
moved from grade 12 to grade 1 1 . Furthermore, during the years for which these data were 
gathered, only student portfolios in the areas of writing and mathematics, and on-demand 
assessments in the areas of reading, mathematics, science, and social studies, were considered in 
calculating accountability index scores. 

The contextual and socioeconomic data regarding the student population of each school 
were obtained from district records. These data include: school enrollment, percent of minority 
students, percent of students qualifying for free or reduced lunch benefits, percent of students 
enrolled in special education, percent mobility (students who withdraw or are added to a school’s 
enrollment in a given year), percent retention (students not promoted to the next grade 1 ), 
suspension days (number of days students are suspended), expulsion days (number of days students 
are expelled), and pupil/teacher ratio. Inspection of district records showed that for individual 
schools, these school-based measures varied little from 1993 to 1995. Therefore, the most recent 
data (1995) from each school were used in all analyses. 

Method 

The data were analyzed in four stages. First, means and standard deviations were computed 
for all measures to determine the extent of variation among the schools. Because of differences in 
the procedures used to calculate the accountability index scores for elementary, middle, and high 
schools, data from each of these levels were analyzed separately at all stages in the analysis. 
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Second, correlation coefficients were computed between all measures to determine the 
degree of linear relationship. Third, multiple regression analyses were conducted in which the 
KIRIS accountability index scores from each year were regressed on the entire group of contextual 
and socioeconomic indices. Separate multiple regression analyses also were conducted using gain 
scores as the dependent variables. 

Finally, contextual and socioeconomic indices were eliminated from each regression model 
in stepwise fashion to produce the most parsimonious models for predicting accountability index 
scores and gain scores. 

Results 

Descriptive statistics are illustrated in Table 1 and show substantial variation among schools 
at all levels on nearly every variable. Exceptions include the expulsion days at the elementary level 
(0 for all schools) and pupil/teacher ratio (nearly the same for schools at each level). On most 
measures, variation is greatest among the elementary schools and least among the high schools. 
Enrollment, for example, ranges from 222 to 884 for elementary schools, from 245 to 990 for 
middle schools, and from 1277 to 1606 for high schools. Similarly, the percent of minority students 
varies from 5% to 72% for elementary schools, from 12% to 45% for middle schools, and from 
16% to 41% for high schools. Accountability index scores show considerable variation as well, 
with 1995 scores ranging from 29.2 to 72.8 for elementary schools, from 38.3 to 75.6 for middle 
schools, and from 36.2 to 56.3 for high schools. 

[Insert Table 1] 
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The results of the correlation analyses are shown in Tables 2-4. These analyses were 
somewhat restricted at the high school level because there are only five high schools in this district. 
Nevertheless, several interesting patterns emerged. 

As expected, many of the contextual and socioeconomic indicators are highly correlated at 
all three levels of schools. Particularly interesting is the relationship between the primary indicator 
of poverty and that of minority enrollment. Correlations between percent of students in a school 
qualifying for free or reduced lunch benefits and percent of minority students are .82, .92, and .96 
for elementary, middle, and high schools, respectively. Also highly correlated at most levels with 
percent of students qualifying for free or reduced lunch benefits are percent of students in special 
education programs, mobility rates, and retention rates. In other words, in this school district, 
schools with high percentages of students from economically disadvantaged backgrounds also tend 
to have more minority students, more students placed in special education programs, more mobility 
among students, and more students retained. 

Also as expected, accountability index scores from the three years are highly correlated. 
This shows that schools with high accountability scores tend to maintain relatively high scores 
while those with low scores remained relatively low. Gain scores, on the other hand, were 
uncorrelated with accountability index scores and negatively correlated across years. In other 
words, schools’ scores on the accountability index had no relation to the gain they were able to 
achieve from one year to next. Unexpectedly, however, the gains schools achieved from 1993 to 
1994 on the accountability index were negatively related to gains they achieved from 1994 to 1995 
(elementary: r = -.54; middle: r = -.40; high: r = -.47). This means that gains from year to year are 
not simply unstable; they are inversely related. In general, the more a school gained one year, the 
less it gained the following year, and vice versa. 
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[Insert Tables 2, 3, & 4] 



Among elementary schools, the correlations between accountability index scores and 
enrollment were positive but not statistically significant (r = +.26 to +.33). Correlations between 
other contextual and socioeconomic variables and accountability index scores, however, were 
generally negative and large in magnitude (r = -.53 to -.80). This means, for example, that as the 
percent of students qualifying for free and reduced lunch benefits in a school increased, 
accountability index scores generally decreased. These trends changed only slightly for middle 
schools, where again the correlations between socioeconomic indices and accountability index 
scores were negative and equally large in magnitude (r = -.60 to -.96). The contextual variables of 
mobility, retention, and suspensions also were negatively correlated with accountability index 
scores at this level (r = -.64 to -.91). 

At the high school level, correlations between contextual and socioeconomic variables and 
accountability index scores were similarly negative and large in magnitude (r= -.66 to -.97). A 
more detailed analysis of the high school data showed, however, that four of the five schools 
clustered closely on nearly all measures while one school was a consistent outlier. This pattern, 
combined with the small number of high schools included in the analysis, may account for the 
unusually high correlations between several variables. 

Interestingly, correlations between the contextual and socioeconomic variables and gain 
scores were consistently small and not statistically significant at all school levels. In other words, 
improvement in accountability index scores appears to be independent of the contextual and 
socioeconomic characteristics of a school’s student population. 



Regression analyses were then conducted, first using the entire set of contextual and 
socioeconomic measures as independent variables, and accountability index scores or gain scores as 
the dependent variable. Predictor variables next were eliminated in stepwise fashion, testing the 
reduction in the R-square value. It was discovered that a single socioeconomic variable, percent of 
students qualifying for free or reduced lunch benefits, provided the most parsimonious model for 
predicting accountability index scores. This is undoubtedly due to the high degree of multi- 
collinearity among the contextual and socioeconomic indicators included in these data. 

Results from the regression analyses showed the percent variance explained (R-squared) in 
accountability index scores for the three years ranged from 51% to 66% for elementary schools, 
from 60% to 81% for middle schools, and from 61% to 78% for high schools. In all cases, these 
values are higher than the R-squared value (42%) obtained in the previously noted research that 
involved a statewide, multiple-choice, basic skills testing program (Guskey & Kifer, 1990). Thus 
the change in assessment format appears not to have altered the relationship between this 
socioeconomic indicator and assessment results. 

It should also be noted, however, for elementary and middle schools the R-square values 
were highest for 1993 results and lowest for 1995 results. At the high school level, the opposite 
trend was noted: the R-square value was lowest for 1993 results and highest for 1995 results. 
Although the differences among these R-square values are not statistically significant, this finding 
provides preliminary evidence that over time, this socioeconomic indicator may be becoming a less 
powerful predictor of accountability index scores for elementary and middle schools. Whether the 
opposite trend for high schools is generalizable or unique to the high schools in this particular 
district is unknown. 
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Regression analyses with gain scores yielded much smaller and nonsignificant R-square 
values at all school levels. This shows that socioeconomic indicators are not powerful predictors of 
the gains schools can achieve on these performance-based student assessments. Results from 
analyses of the most current accountability index scores (1995) are illustrated in Table 5. 

[Insert Table 5] 

These mixed findings are surprising but generally optimistic. They indicate that like the 
results from the basic skills testing program used formerly in the Commonwealth, socioeconomic 
indicators are powerful predictors of a school’s initial achievement standing. Such indicators, 
however, do not predict the progress or improvement a school can make. The contextual and 
socioeconomic variables considered in this study were unrelated the improvements schools made, 
measured in terms of results on performance-based student assessments. The negative relationship 
between the gains made by schools in two consecutive years, however, remains puzzling. 

Conclusions and Implications 

The results from this investigation show that although socioeconomic indicators explain 
much of the variation in elementary, middle, and high schools’ initial level of achievement in a high- 
stakes, performance-based student assessment program, they do not explain the variation in score 
improvements made by schools in this school district. In other words, the contextual characteristics 
of a school and the socioeconomic characteristics of the school’s student population are unrelated 
to the magnitude of improvement attained. This is a very positive finding from the perspective of 
advocates of performance-based assessment systems who have hoped this change in assessment 



content, format, and procedures would result in fair and instructionally valid measures of student 
learning. These results also suggest that this high-stakes, performance-based assessment system 
may be prompting schools at all levels to make changes that lead to improvements in the 
performance of their students. 

At the same time, there are two aspects of these results that deserve special attention. First 
is the difference in trends for elementary and middle schools versus high schools. Among the 
elementary and middle schools, the relationship between socioeconomic indicators and student 
performance results is smaller in magnitude and appears to be diminishing. Among the high 
schools, however, this relationship is larger in magnitude and appears to be increasing. This finding 
is particularly troubling if one considers the probable impact of school dropouts on the assessment 
results. 

Approximately 22 percent of the high school students enrolled in the ninth grade in this 
school district fail to graduate within four years. Those students who dropout or otherwise fail to 
graduate typically have low levels of achievement and poor academic records. They also include 
disproportionately large numbers of economically disadvantaged and minority students. The effect 
of dropouts, therefore, is to reduce the variance in both achievement measures and socioeconomic 
indicators. This reduction in variance, in turn, should reduce the magnitude of the correlation 
between these measures. 

In this particular instance, however, the relationship between socioeconomic indicators and 
achievement measures is greater in magnitude at the high school level than at the elementary and 
middle school levels, and appears to be increasing. The nature of these data, particularly the 
divergence of one of the high schools from the other four on many of the measures, confounds 
efforts to provide precise explanations. Nevertheless, this finding clearly calls for an investigation 
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in this district of possible unintended consequences stemming from formal or informal tracking 
practices, or other inequitable opportunities for students to learn (Winking & Bond, 1996). 

The second aspect of these results deserving special attention is the negative correlation 
between accountability index score gains from one year to the next for all school levels. In other 
words, the gains schools achieved from 1993 to 1994 on the accountability index are negatively 
related to gains achieved from 1994 to 1995. Some have suggested this negative correlation is an 
indication that the gains are spurious or artificial. Others argue it may be due to differences in the 
quality or difficulty of certain aspects of the assessment system from year to year (Kentucky 
Institute for Education Reform, 1995). A closer examination of school-by-school results yields still 
another plausible explanation. 

The Kentucky Instructional Results Information System, with its accompanying 
accountability system, is based on the assumption “What you test is what you get.” That is, in a 
high stakes environment where rewards or sanctions are tied to test performance, teachers will alter 
their instruction to match the content and format of the tests. Although this is generally true with 
regard to basic skills testing, current evidence indicates it may not be true when more authentic, 
performance-based assessments are used (Linn, 1993). Most teachers have scant knowledge, 
personal background, experience, or formal training in performance assessments, or in how to use 
them as instructional tools (Rothman, 1995). In addition, the majority of teachers indicate they do 
not have sufficient time to administer and score such assessments, nor do they have access to 
appropriate and well-aligned instructional materials (Vitali, 1993). 

To make these substantial changes in their instructional practices, most teachers need 
extended time and sustained, high quality professional development (Guskey, 1994b). But instead 
of providing such time and resources, pressure for immediate improvement in scores has prompted 
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many schools in Kentucky to develop professional development programs that focus narrowly on 
the particular assessment formats and scoring procedures included in KIRIS (Oldham, 1994). A 
recent study showed, for example, that all surveyed principals reported encouraging teachers to use 
special test-preparation materials and to teach explicit test-taking skills (Koretz, Barron, Mitchel, & 
Stecher, 1996). Consequently, teachers teach their students precise strategies for tailoring 
responses to specific scoring rubrics. Although this generally leads to improved scores, such 
improvements are typically modest and short-lived (Guskey & Oldham, 1997). 

In examining school-by-school data, it appears about one-third of the schools in this district 
may have improved their assessment results in the first year by teaching students these specific 
response strategies. As a result, they experienced a first year increase in their accountability index 
scores. In the second year, however, improvements leveled off. Another third of the schools may 
not have provided this focus until the second year. Their results were relatively flat during the first 
year but saw an increase in year two. The final third of the schools apparently did not provide this 
focus or any other meaningful alternative to improve results. Hence, they saw little overall gain. 

The gains achieved by schools in one year were not, therefore, followed by a loss the next 
year, or vice versa. Rather, it is simply that among the schools in this school district, significant 
gains are typically followed or preceded by a period of relatively flat improvement. Perhaps this is 
because significant gains are difficult to sustain from year to year. Perhaps steady improvements 
require more time and resources than have been provided. Or perhaps, as suggested, the gains 
evidenced thus far have been attained simply by teaching students specific response strategies, and 
the hoped-for improvements in teaching practices that will lead to greater depth in students’ 
understanding have yet to be realized. Additional studies exploring these issues are clearly needed. 



especially efforts that might include the disaggregation of accountability indices to consider changes 
in results from year to year in specific subject areas. 

Because the results from this study are based on data gathered from a single school district, 
generalizations are limited. Nevertheless, it does provide preliminary evidence on the relationship 
between contextual variables and socioeconomic indicators, and school-level results from a 
performance-based student assessment program. Although performance assessments clearly do not 
provide a solution to all of the problems schools face in their efforts to improve the quality of 
education provided students, they may be moving us in the right direction. 

Endnotes 

1 The percent retention for elementary schools is based only on students retained in grades 4 and 5. 
From kindergarten through grade 3 students are enrolled in a “non-graded primary school 
program,” which is mandated as part of the Kentucky Education Reform Act. Under this program, 
students who are “given another year” in the primary grades are not counted in determining a 
school’s retention rate. 
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Descriptive Statistics for Elementary, Middle, and Secondary Schools 
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